Super-resolution (SR) methods aim to recover new high-resolution (HR) information beyond the Nyquist frequency of the low-resolution (LR) image. SR methods are applicable in relation to HDTV, video communication, video surveillance, medical imaging, and other applications. Recently, example-based SR (also commonly referred to as “hallucination”) that reconstructs HR image from one single LR input image has emerged as a promising technology because it can overcome some limitations of the classical multi-image super-resolution methods and can be implemented with lower computation and memory costs.
Example-based SR methods assume that the missing HR details can be learned and inferred from a representative training set or the LR image itself. For example, an image predication model may rely on a database of example patches with the low-frequency and the residual higher frequency bands built using a training image set. The LR image is partitioned into a plurality of overlapping patches. For each patch, a search is performed in a database according to the low-frequency component of the example patches to identify the corresponding high frequency band for reconstruction. Other representative approaches of this kind include Kernel Ridge Regression based SR methods, sparse coding based SR methods, etc. These types of SR approaches are capable of producing plausible fine details across the image; however, a lack relevant examples in the database often causes noisy images and irregularities along curved edges. Moreover, the use of larger databases is more time and memory consuming and effective hardware implementation may be a challenge.
Another example-based SR method employs the self-similarity characteristics of the image, i.e. based on the observation that small patches in a natural image tend to redundantly recur many times inside the image. In one prior method both the patch recurrence within the same image scale and across different coarser image scales were employed to recover information among subpixel misalignments and implicit low-resolution/high-resolution pairs. A single unified computational framework was used to combine these two SR approaches and an Approximate Nearest Neighbor (ANN) algorithm is employ to accelerate the patch searching. In another method, a dictionary of lower-resolution/high-resolution pairs was built online using the image pyramid of the input LR image and was then refined using group sparsity constraints. HR images were reconstructed using an ΔNN search in the dictionary. In still another method, a local self-similarity assumption on natural images were followed, so that patches were extracted from localized regions as small as 10×10 rather than the whole input image. This method reduced the nearest-patch search time considerably, without compromising quality in most images. In still another method, a non-local back-projection method was proposed, where a local search in a small window was employed to recover non-local redundancies and suppress “jaggy” and “ringing” artifacts. As opposed to small upscale steps, for example 4×4 to 5×5, in other aforementioned algorithms, the image is typically enlarged two times in each direction, so that it could reach the target image enlargement ratio by just building one or two layers of reconstructed images.
Such example-based SR algorithms are often computationally intensive, because for each pixel or patch the methods require searching the high-resolution counterpart in a database/dictionary, an image pyramid or a small image region. Although efforts have been made to reduce the computational complexity, it is still a major challenge to the commercial application of the SR technology.